Multi touch with multi haptics

ABSTRACT

Methods and systems for processing touch inputs are disclosed. The invention in one respect includes reading data from a multi-touch sensing device such as a multi-touch touch screen where the data pertains to touch input with respect to the multi-touch sensing device, and identifying at least one multi-touch gesture based on the data from the multi-touch sensing device and providing an appropriate multi-haptic response.

CROSS REFERENCE TO RELATED APPLICATIONS

This patent application takes priority under 35 U.S.C. 119(e) to U.S.Provisional Application Ser. No. 61/140,519 entitled MULTI TOUCH WITHMULTI HAPTICS by Burrough et al., filed Dec. 23, 2008 which isincorporated by reference in its entirety for all purposes.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to providingmulti-touch/multi-haptic systems and methods.

2. Description of the Related Art

Multi-touch devices have advantages over conventional single pointsensing touch devices in that they can distinguish more than one object(finger) in contrast to single point devices that are simply incapableof distinguishing multiple objects. In most cases, multi-touch devicesmonitor a sensing surface for a touch or near touch, and when a touchoccurs determines the distinct areas of contact and identifies thecontacts via their geometric features and geometric arrangement. Onceidentified or classified, the contacts are monitored for variousmotions, actions or events. The contacts and motions thereof are thenconverted into inputs for controlling some aspect of an electronicdevice.

Multi-touch devices can be embodied in various forms including but notlimit to standard touch pads, large extended palm pads, touch screens,touch sensitive housings, etc. Furthermore, multi-touch devices can beplaced in various electronic devices including but not limited tocomputers such as tablet computers, laptop computers, desktop computersas well as handheld computing devices such as media players (e.g.,music, video, games), PDAs, cell phones, cameras, remote controls,and/or the like. The multi-touch devices can also be placed on dedicatedinput devices such as touch screen monitors, keyboards, navigation pads,tablets, mice, and the like. Essentially, multi-touch devices can beapplied to any surface, and can be found in any consumer electronicproduct that requires inputs.

Since multi-touch devices provide a number of inputting operations at asingle location (input surface), inputting with multi-touch devices canbe very efficient. The user can maintain their hand(s) at themulti-touch surface without having to move their hand(s) to addressother input devices. For example, conventional systems typically includea keyboard and a separate mouse. In order to use the mouse, the usermust move their hand from the keyboard and onto the mouse. In order tokeyboard efficiently (both hands), the user must move their hand fromthe mouse to the keyboard. This inputting sequence is very inefficient.For one, only one device can be used effectively at a given time. Foranother, there is wasted time between each inputting step. In contrast,with multi-touch surfaces the user can generate both static commands(e.g., keyboarding) and manipulative commands (e.g., tracking) from thesame location and at the same time. The user therefore does not have tomove their hands to perform different inputting tasks. The user simplyprovides different chords or finger motions to generate a number ofinputs either sequentially or simultaneously. In one example, the usercan provide key commands with taps at specific locations of themulti-touch surface while allowing tracking from all locations of themulti-touch surface.

However, research has shown that providing the multi-touch surface withthe ability to provide physical (haptic) feedback makes the multi-touchexperience even more efficient and realistic to the user. For example,physical keyboards provide a physical indication (a bump, for example)indicative of the home key. This physical sensation can not be providedby a conventional multi-touch system thereby forcing the user tovisually locate the home key thereby making keyboard use less efficientand fatiguing. However, by providing a physical facsimile of the homekey bump using an actuator that provides a physical sensation to theuser providing an approximate representation of the bump, the user'sexperience of the MT keyboard (and any multi-touch system for thatmatter) can be more realistic and therefore more enjoyable.

Therefore, a system that enhances the multi-touch experience byincorporating a corresponding physical response(s) is described.

SUMMARY OF THE INVENTION

The invention relates, in one embodiment, to an apparatus and method forproviding multi-touch haptic feedback. The apparatus includes, at least,a touch pad having a touch sensitive surface arranged to receive a userprovided multi-touch event associated with at least two differentlocations on the touch sensitive surface, a multi-touch detectionmechanism operatively coupled to the touch sensitive surface thatdetects the multi-touch event and generates a corresponding amulti-touch signal, and a plurality of haptic feedback devicesoperatively coupled to the multi-touch detection mechanism and the touchsensitive surface cooperatively arranged to concurrently provide tactilefeedback at each of the at least two different locations on the touchsensitive surface in response to the multi-touch signal wherein thetactile feedback at each of the at least two different locations arediscreet from one another. When the multi-touch signal indicates thatthe multi-touch event is a dynamic multi-touch event indicating a changein the multi-touch event, then the tactile feedback at each of the atleast two different locations is updated to reflect the change in themulti-touch event.

It should be noted that in some cases the tactile feedback event can bedifferent for each of the at least two different locations.

The invention relates, in another embodiment, to a multi-touch hapticmechanism. The multi-touch haptic mechanism includes, at least, a touchpad having a touch sensitive surface arranged to detect a user touchevent at substantially any location on the touch sensitive surface and aplurality of independent haptic devices operatively coupled to the touchsensitive surface each providing a corresponding type of tactilefeedback thereby providing a tactile feedback at substantially anylocation on the touch sensitive surface at which the user touch eventhas occurred, wherein each of the plurality of independent hapticdevices only responds to the user touch event in one or more associatedregions of the touch sensitive surface. When at least two of theplurality of independent haptic devices cooperate to provide a type ofhaptic response that is different than that type provided by either ofthe at least two independent haptic devices separately.

The invention relates, in another embodiment, to an integrated devicearranged to act as both a force sensing device and a haptic feedbackdevice. The device includes, at least, a touch sensitive surface, acontroller unit, and a mechanical actuator coupled with the controllerunit and the touch sensitive surface. The integrated device acts as theforce sensing device by generating an output voltage in directproportion to a force applied to the mechanical actuator by a usertouching the touch sensitive surface, sensing the output voltage by thecontroller unit and converting the sensed output voltage to anindication of the applied force. Only when the sensed output voltageexceeds a voltage threshold level does the integrated device act as thehaptic feedback device by halting the sensing of the output voltage bythe controller unit activating the mechanical actuator by the controllerunit, wherein the activated mechanical actuator imparts a physical forceto the touch sensitive surface that results in a vibro-tactile response(subcutaneous tissue activated) felt by the user commensurate with theforce applied by the user.

The invention relates, in another embodiment, to an electronic device.The electronic device includes, at least, a touch pad having a touchsensitive surface arranged to process a user touch event and a pluralityof haptic feedback devices each of which is operatively coupled to thetouch sensitive surface and each responding to the user touch event onlyin a specific region of the touch sensitive surface and arranged toprovide tactile feedback singly or in combination with others of theplurality of haptic feedback devices in response to the user touchevent. When the touch sensitive regions of at least two of the pluralityof haptic devices overlap, if the user touch event occurs in theoverlapping region, then the at least two haptic devices cooperate toprovide a combined haptic feedback response based upon the location inthe overlapping region of the user touch event.

The invention relates, in another embodiment, to an electronic device.The electronic device includes, at least, a touch pad having a touchsensitive surface arranged to receive a user touch event provided by auser, a controller coupled and in communication with the touch padarranged to at least analyze the user touch event and/or a state of thetouch pad and based upon the analysis provide a user touch event signalin response to the user touch event, and at least one haptic deviceoperatively coupled to the controller arranged to receive the user touchevent signal, wherein the at least one haptic device responds to theuser touch event signal by providing an appropriate haptic feedbackresponse to the user based upon the analysis provided by the controller.

In one embodiment, the touch sensitive surface is arranged to receivedifferent types of user touch events each being characterized by anamount of pressure applied on the touch sensitive surface by a user andat least one haptic device operatively coupled to the touch sensitivesurface arranged to provide a specific type of tactile feedbackcorresponding to the amount of pressure applied to the touch sensitivesurface by the user.

It should be noted that in each of the embodiments described above, themethods can be implemented using a touch based input device such as atouch screen or touch pad, more particularly a multi-touch touch basedinput device, and even more particularly a multi-touch touch screen. Itshould also be noted that the gestures, gesture modes, gestural inputs,etc. can correspond to any of those described below in the detaileddescription. For example, the gestures can be associated with zooming,panning, scrolling, rotating, enlarging, floating controls, zoomingtargets, paging, inertia, keyboarding, wheeling, and/or the like.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be readily understood by the following detaileddescription in conjunction with the accompanying drawings, wherein likereference numerals designate like structural elements, and in which:

FIGS. 1A-1E are a series of block diagrams of a system, in accordancewith one embodiment of the present invention.

FIGS. 2A-2B shows a multi-point multi-haptic system having a multi-touchsurface that incorporates a plurality of haptic devices in accordancewith an embodiment of the invention.

FIG. 3 shows a schematic diagram of a representative piezo-electrichaptic assembly.

FIG. 4 shows a schematic diagram of the haptic assembly shown in FIG. 3configured to act as a pressure sensor.

FIG. 5 shows a flowchart detailing a process in accordance with anembodiment of the invention.

FIG. 6 shows display device displaying representative haptic active GUIelements in accordance with an embodiment of the invention.

FIG. 7 shows representative GUI button elements in accordance with anembodiment of the invention.

FIGS. 8A-8B shows representative GUI button element and associatedhaptic profile in accordance with an embodiment of the invention.

FIGS. 9A-9B shows a representative slider element and associated hapticprofile in accordance with an embodiment of the invention.

FIGS. 10A-10B shows a feature edge detection system in accordance withan embodiment of the invention.

FIG. 11 is a diagram of a zoom gesture method 1100 in accordance with anembodiment of the invention.

FIGS. 12A-12H illustrates a display presenting a GUI object in the formof a map of North America with embedded levels which can be zoomed.

FIG. 13 is a diagram of a GUI operational method in accordance with oneembodiment of the present invention.

DETAILED DESCRIPTION OF SELECTED EMBODIMENTS

Reference will now be made in detail to selected embodiments an exampleof which is illustrated in the accompanying drawings. While theinvention will be described in conjunction with a preferred embodiment,it will be understood that it is not intended to limit the invention toone preferred embodiment. To the contrary, it is intended to coveralternatives, modifications, and equivalents as can be included withinthe spirit and scope of the invention as defined by the appended claims.

The invention relates to multi-touch haptic feedback. Multi-touch hapticfeedback refers to haptic techniques capable of providing multiple anddiscretely located haptic sensations across a surface. The haptic systemcan for example include a plurality of haptic nodes, each of which iscapable of issuing vibro-tactile sensations (at the same time ordifferent times and with the same intensity or different intensity). Thehaptic nodes can for example be configured in a matrix or array. In oneembodiment, the haptic nodes are mapped to touch sensing nodes. Eachtouch sensing node can be assigned one or more haptic nodes. The hapticnodes are typically proximate the touch sensing nodes to which it hasbeen assigned.

In one embodiment, the touch sensing surface is a multi touch surfacethus making a multi touch multi-touch haptic device. In so doing hapticfeedback can be provided that indicates information about a multi touchevent. For example, the surface under a moving finger can be actuatedwhile the surface under the non-moving finger remains static. In anotherexample, the surface under the moving finger is actuated concurrentlywith a signal being passed to the other finger indicating that a multitouch action is occurring. In this way, the signals taken together canindicate the nature of the underlying action being taken by the user.For example, if an object (such as an image) is being expanded orreduced in size a larger/more intense signal could be generated (eitherby increasing frequency or amplitude). It is also contemplated thatisolated feedback can be used to provide an on-screen click-wheel orother such user input where the touch screen is used to simulate the“clicks” of the click wheel both audibly and via tactile feedback.

The described embodiments generally pertain to gestures and methods ofimplementing gestures with associated physical feedback with touchsensitive devices. Examples of touch sensitive devices include touchscreens and touch pads. One aspect of the invention describes a touchsensitive input device able to recognize at least two substantiallysimultaneously occurring gestures using at least two different fingersor other objects (hereinafter referred to as a multi-touch event). Thetouch sensitive input device communicates with an array of hapticfeedback devices (also referred to as haptic actuators) each arranged toprovide haptic feedback according to a haptic profile in response to amulti-touch event. In another aspect of the invention, each fingerreceives different haptic feedback (multi-haptic) depending upon thelocation on the touch sensitive input device each finger is placed. Inanother aspect of the invention, a compound haptic feedback can beprovided that combines the output from at least two different hapticactuators to form the compound response that is different from thatprovided by the two originating haptic actuators. In another embodiment,an integrated device is described that can act as both a force sensingdevice and a haptic feedback device. In still another embodiment, ahandheld portable device is described having a housing and a userinterface are acoustically isolated from each other. In this way, thehousing and user interface and having non-interfering and independenthaptic responses.

These and other aspects of the invention are discussed below withreference to FIGS. 1-13. However, those skilled in the art will readilyappreciate that the detailed description given herein with respect tothese figures is for explanatory purposes as the invention extendsbeyond these limited embodiments.

FIG. 1A-1E are block diagrams of a representative electronic device orsystem 100, in accordance with one embodiment of the present invention.Electronic device 100 can correspond to a computer (such as a desktopsor laptops) as well as small form factor electronic devices that caninclude portable consumer electronic products such as cell phones, PDA,media players and/or the like. As such, portable electronic device 100can be sized for one-handed operation and placement into small areassuch as a pocket. Portable electronic device 100 can process data andmore particularly media such as audio, video, images, etc. As such, theportable electronic device 100 can correspond to a music player, gameplayer, video player, personal digital assistant (PDA), such as, forexample, an iPod™, an iPod Nano™, an iPod Shuffle™, an iPod™ Touch or aniPhone™ available by Apple Inc. of Cupertino, Calif. In some cases,portable electronic device 100 can communicate wirelessly (with orwithout the aid of a wireless enabling accessory system) and/or viawired pathways (e.g., using traditional electrical wires).

Portable electronic device 100 includes a housing 102. Housing 102 canbe formed of any number of materials including for example plastics,metals, ceramics and the like. In one embodiment, housing 102 can beformed of stainless steel in order to provide an aesthetic and appealinglook and feel as well as provide structural integrity and support forall sub-assemblies installed therein. Housing 102 can define a cavityconfigured to at least partially enclose any suitable number ofoperational electronic components 104 used by portable electronic device100 to carry out its intended functions. Operational electroniccomponents 104 can include processor 106 that can operate (inconjunction with an operating system) to execute computer code andproduce and use data. Processor 106 can be implemented on a single-chip,multiple chips or multiple electrical components. For example, variousarchitectures can be used for the processor 106, including dedicated orembedded processor, single purpose processor, controller, ASIC, and soforth. The operating system, other computer code and data can residewithin a memory 108 that can be operatively coupled to processor 106. Byway of example, memory 108 can include Read-Only Memory (ROM),Random-Access Memory (RAM), flash memory, hard disk drive and/or thelike. Operational components 104 can also include a number ofinput/output (I/O) devices 109. Such devices can include audio outputdevices such as headphone jacks, data ports (such as I.E.E.E. 1392compliant, USB, etc.), and so on.

Portable electronic device 100 can also include a user interface 110that can operate to both receive user inputs and provide information toa user. In the described embodiment, user interface 110 can includedisplay device 112 that can be operatively coupled to processor 106 byway of bus 114. Display device 112 can correspond to any known displaytechnology such as a plasma, LCD, or an organic light emitting diode(OLED). It should be noted that in the embodiment shown in FIGS. 1A-1E,display device 112 is integrated with the electronic device 100.However, display device 112 can also be configured as a componentseparate from portable electronic device 100 in which case displaydevice 112 would be considered a peripheral device that can be coupledto portable electronic device 100 by way of a wired connection (such asa peripheral bus or cable) or a wireless connection such as IR, RF,Bluetooth or the like (among others).

In some cases, display device 112 presents graphical user interface(GUI) 116 on display device 112. GUI 116 can provide an easy to useinterface between a user of portable electronic device 100 and theoperating system or application running thereon. Generally speaking, GUI116 iconically represents programs, files and operational options withgraphical images. The graphical images can include windows, fields,dialog boxes, menus, icons, buttons, cursors, scroll bars, etc. Suchimages can be arranged in predefined layouts, or can be createddynamically to serve the specific actions being taken by a user. Duringoperation, the user can select and activate various graphical images inorder to initiate functions and tasks associated therewith. By way ofexample, a user can select a button that opens, closes, minimizes, ormaximizes a window, or an icon that launches a particular program. GUI116 can additionally or alternatively display information, such as noninteractive text and graphics, for the user on the display device 112.

As shown more clearly in FIG. 1B in a side view perspective of device100, user interface 110 can include protective layer 120 disposed on topof display device 112. In this way, protective layer 120 can be used asprotective top layer of transparent or semitransparent material (clear)thereby affording display device 112 protection from potentiallydamaging external insults caused by, for example, sharp objects,dropping, and so on and yet still allow any image presented by displaydevice 112 to be clearly viewed by a user. Protective layer 120 can beformed of many well known transparent materials such as glass (e.g.,referred to as cover glass), and more particularly highly polishedglass. It should be appreciated, however, that other transparentmaterials (or at least translucent materials) such as clear plastic mayalso be used. In some embodiments, protective top layer 120 can beacoustically isolated from housing 102 using, for example, acousticisolation buffers 121. By acoustically isolating housing 102 andprotective top layer 120 from each other, it is possible to provideseparate and independent haptic responses, one directed at housing 102and another directed at protective top layer 120 without interferingwith each other. For example, it may be desirable to provide one type ofhaptic response at protective layer 120 and another type haptic responseat housing 102 at the same time or at a different time independent ofeach other or in some cases one being the result of or related to theother.

User interface 110 can be touch sensitive suitable for receiving one ormore user touch events by which information can be passed between theuser and the portable electronic device 100. In some cases, the one ormore inputs in the form of user touch events can be substantiallysimultaneously received (e.g., multi-touch). In these embodiments, userinterface 110 is rendered touch sensitive by means of a touch sensinglayer 122 that can be disposed below protective layer 120 such thattouch sensing layer 122 is between protective layer 120 and the displaydevice 112. This arrangement can be accomplished by, for example,applying touch sensing layer 122 to display device 112 or by applyingtouch sensing layer 122 to protective layer 120 using any number ofattachment processes, such as printing, depositing, laminating, etc.Touch sensing layer 122 generally includes at least one touch sensingdevice 124 configured to detect an object in close proximity to orexerting pressure on an upper surface 126 of protective layer 120. Inkeeping with the wide applicability of the invention, sensing device 124can be widely varied and can be configured to activate as the fingertouches the upper surface 126. In the simplest case, an electricalsignal is produced each time a finger (or other appropriate object)passes a sensor. The number of signals in a given time frame mayindicate location, direction, speed and acceleration of the finger onthe touch sensitive portion, i.e., the more signals, the more the usermoved his or her finger.

Touch sensing layer 122 can be configured to act as a multi-touch inputdevice. Multi-touch input devices have several advantages overconventional single point devices in that they can distinguish more thanone object (finger). As a multi-touch input device, touch sensing layer122 can distinguish a wide range of different gestures. By way ofexample, the gestures may be single point or multi-touch gestures,static or dynamic gestures, continuous or segmented gestures, and thelike. It should be noted that single point gestures are those gesturesthat are performed with a single contact point, e.g., the gesture isperformed with a single touch as for example from a single finger, apalm or a stylus. Multi-touch gestures are those gestures that can beperformed with multiple points, e.g., the gesture is performed withmultiple touches as for example from multiple fingers, fingers andpalms, a finger and a stylus, multiple styli and/or any combinationthereof. Static gestures are those gestures that do not include motion,and dynamic gestures are those gestures that do include motion.Continuous gestures are those gestures that are performed in a singlestroke, and segmented gestures are those gestures that are performed ina sequence of distinct steps or strokes.

Touch sensing device 124 can be sensitive to at least one of severalindependent and spatially distinct touch sensing nodes or regions 128.Touch sensing device 124 can positioned throughout touch sensing layer122. Sensing regions 128 are typically not visible to the user anddispersed about protective layer 120 with each sensing region 128representing a different position on surface 126 in coordination withthe locations of sensing device 124. Sensing regions 128 can bepositioned in a grid or other such array where each sensing region 128can generate a signal in response to a user touch event in proximitythereto. Typically, the number of fingers in contact with the surface126 can indicate an input mode. For example, a single touch by onefinger can indicate the desire to perform tracking, i.e., pointer orcursor movements, or selections. On the other hand, multiple touchesusing, for example, a group of fingers can indicate the desire toperform gesturing. The number of fingers in the group used forimplementing gesturing may be widely varied. By way of example, twofingers can indicate a first gesture mode, three fingers may indicate athird gesture mode, etc. Alternatively, any number of fingers, i.e.,more than one, may be used for the same gesture mode, which can includeone or more gesture controls.

The number and configuration of sensing nodes 128 can be widely varied.The number of sensing nodes 128 generally depends on the desiredsensitivity as well as the desired transparency of touch sensing layer122. For example, more nodes or sensing nodes generally increasessensitivity, but may reduce transparency (and vice versa). With regardsto configuration, sensing nodes 128 generally map touch sensing layer122 into a coordinate system such as a Cartesian coordinate system, apolar coordinate system or some other coordinate system. When aCartesian coordinate system is used (as shown), sensing regions 128typically correspond to x and y coordinates. When a polar coordinatesystem is used, the sensing nodes typically correspond to radial (r) andangular coordinates (θ). In this way, touch sensing layer 122 can trackmultiple objects, such as fingers, which rest on, tap on, or move acrossan upper surface 126 of protective layer 120. In this way, a user canperform several touch initiated tasks at the same time. For example, theuser can select an onscreen button with one finger, while moving acursor with another finger. In addition, a user can move a scroll barwith one finger while selecting an item from a menu with another finger.Furthermore, a first object can be dragged with one finger while asecond object can be dragged with another finger.

In the simplest case, a touch event T is initiated each time an object,such as a user's finger, is placed on upper surface 126 over, or inclose proximity to, sensing region 128. Pressure generated by touchevent T is transmitted through protective layer 120 at sensing region128 to sensing device 124. In response to the pressure applied by theuser during touch event T, sensing device 124 generates touch signal S₁(and any other signal consistent with a multi-touch event). Touch signalS₁ can be monitored by an electronic interface (not shown) and passed toprocessor 106. Processor 106, in turn, can convert the number,combination and frequency of the signal(s) S into touch informationT_(info) that can include location, direction, speed and accelerationinformation of touch event T. Processor 106 can then pass touchinformation T_(info) to micro-controller 132. Although micro-controller132 is shown as a component separate from processor 106, it iscontemplated that functions carried out by micro-controller 132 can infact be performed by processor 106.

Micro-controller 132 can use touch information T_(info) to query hapticdata base 134 that includes a number of pre-determined haptic profileseach of which describes a specific haptic response H_(x) in terms ofduration of response, type of vibro-tactile response, strength ofresponse, etc. A particular haptic profile includes a set ofinstructions that cause microcontroller 132 to activate at least hapticactuator 136. Haptic actuator 136, in turn, creates haptic responseH_(x). In this way, the response of haptic actuator 136 can becontrolled in real time by microprocessor 132 by establishing theduration, strength, type of vibro-tactile response H_(x). Furthermore,the visual information presented on display 112 and the correspondingtactile response can be closely linked. For example, if the location oftouch event T coincides with a visual display of a button icon generatedby display device 112, the corresponding haptic response provided byhaptic actuator 136 can have a haptic profile H_(button) consistent withthat of a dome button.

In some embodiments it may be desirable to associate each sensing node128 with one or more corresponding haptic actuators. For example,sensing node 128 can trigger haptic actuator 136 and/or 140 or even 142independent of each other or in concert. Accordingly, sensing nodes 128can be arranged in such a way as two or more haptic actuators cancooperate to produce a compound haptic effect. It should be noted thatan effective range R (distance over which a particular haptic actuatorcan be felt) for each of the haptic actuators can be based upon manyfactors such as intrinsic nature of the haptic actuator (i.e.,mechanical vs EPAM), the damping properties of protective layer 120, theharmonic frequency of the device 100, and so on.

As shown in FIGS. 1C-1E, touch event T can result in multiple hapticactuators 136 being activated each being driven by a separate set ofinstructions based upon different haptic profiles. For example, as shownin FIG. 1C, haptic actuator 136 can respond with haptic effect H₁whereas haptic actuator 140 can respond with haptic effect H₂ (or H₁ forthat matter). In some cases as shown in FIG. 1D, a compound hapticeffect H_(compound) can be generated by providing the same or differenthaptic profiles to at least two different haptic actuators whichinterfere with each other (either constructively or destructively) toform compound haptic effect H_(compound). Still further, as shown inFIG. 1E, due to the fact that housing 102 and user interface 110 areacoustically isolated from each other, haptic actuator 142 can be usedto provide a haptic response directed at housing 102 independent of anyhaptic response or responses directed at user interface 110. Forexample, touch event T can cause processor 106 to direct microcontroller132 to instruct haptic actuator 142 to produce haptic effect H_(housing)(such as vibrate housing 102) at the same time as instructing hapticactuators 136 (and/or 140) to generate a haptic response H₁ and/or H₂.In this way, a user will feel housing 102 vibrate independent of anytactile response emanating from display interface 110 (and moreparticularly protective layer 120) thereby increasing the amount andvariety of information that can be provided to (and by) the user. Inthis way, user interface 110 using display device 112 and hapticactuator 136 can provide both visual and tactile information to theuser.

It should be noted that haptic actuator 136 can be formed as a small andthin chip and thus can be installed into the mobile apparatus such assmall form factor/handheld devices such as cell phones, media players,and the like and can be electro-polymer based, piezo-electric based, orany combination thereof. In this way, user interface 110 using displaydevice 112 and haptic actuator 136 can provide both visual and tactileinformation to the user. Furthermore, the components of the device 100can be chosen to provide more effective haptic sensations. For example,if haptic actuator 136 oscillates at close to a natural frequency of themechanical system (including the actuator itself), then stronger forcesand more effective haptic sensations can be output. Accordingly, themass (as well as the spring constants of the system 100) can be selectedto provide a desirable low natural frequency, such as about 120 Hz orless, which tends to cause effective haptic sensations. It should alsobe noted that multiple haptic actuators can be driven in unison forstronger haptic effects or at different times to provide sensations atparticular locations of surface 126.

One of the advantages of the invention lies in the fact that therelationship between a touch event or a class of touch events andcorresponding haptic response can be dynamic in nature. By dynamic it ismeant that although specific haptic profiles H stored in haptic profiledata base 134 remain static, the haptic profile (or profiles) used torespond to a particular touch event T can be varied based upon anynumber of factors. For example, if touch event T coincides with aparticular GUI icon displayed by display device 112 (such as the buttonicon described above), the corresponding haptic response H_(x) can varydepending upon not only the icon displayed but also the location onsurface 126 of touch event T (i.e., T(x)), any finger motion(∂T/∂x,∂T/∂y), any pattern of lifting a finger and placing it back downon surface 126 (i.e., (ΔT/Δt) such as multiple clicks in the case of abutton icon, and so on. For example, vibrations can be adjusted based ona change in touch characteristics (i.e., speed, direction, location,etc.) whereas different vibrations can signify different events orregions. Furthermore, vibrations can be mapped to animation effectsoccurring on display 112 (rubber band, bounce etc.) or be based onlocation in the form of an intensity map on the surface 126.

Haptic actuators associated with system 100 can be arranged in groups.For example, one such group can include a primary haptic actuatorarranged to provide a vibro-tactile response and at least one secondarydevice selected from piezo-ceramic element or EPAM arranged to provideanother type response. In this way, the primary haptic actuator can beutilized to convey a status (e.g., incoming call) as well as passiveresponses to touch events such as location of input areas (e.g., virtualbuttons) while the secondary device can be utilized to convey activeresponses to touch events such as signifying that an action was taken(e.g., button click). In some cases, a second secondary haptic devicecan be used to convey other active or passive responses. Some hapticdevices (such as the secondary ones mentioned above) can be better atsimulating mechanical clicks (e.g., dome switch) making the userexperience similar to what they are used to in legacy devices thatutilize mechanical inputs (switches) instead of electrical inputs (touchsensing).

By judiciously selecting appropriate materials, haptic response ofdevice 100 can be adjusted based upon the acoustic properties of thematerials that go into creating device 100. Protective layer 120 can beformed from a material with localized damping properties such thatvibro-tactile sensations in one quadrant are not substantially felt inanother. For example, protective layer 120 can be glass or othertransparent (or at least translucent) material having natural dampingproperties suitable for small form factor devices having small areadisplays. Generally speaking, substantially all of protective layer 120can be provided with haptic sensations as a single unitary member,however, it is possible to provide for individually-moving portions ofprotective layer 120 by providing each portion with their own hapticactuator where the haptic responses can be limited to particular regionof surface 126 with effective range R as shown in FIG. 2A. In this way,haptic sensations can be provided for only a particular portion ofprotective layer 120 using the natural damping factor of glass to limitthe effective range of the actuator. Moreover, the properties ofprotective layer 120 can be selected such that vibro-tactiletransmissions from at least two haptic actuators can interfere with eachother, either constructively or destructively, in order to provide acompound haptic sensation.

Although the resolution can be widely varied, in one embodiment, sensingregions 128 are placed coarsely about the surface. By way of example,sensing regions 128 can be placed at several quadrants (groups of touchsensors). As illustrated in FIG. 2B, a surface 126 (as a rectangle) caninclude about n actuators with a concomitant number of sensing regions128 in order to create vibro-tactile sensations within about n quadrantsof the rectangular oriented touch surface. For example, if n=6, thequadrants can include upper right (I), upper left (II), center left(III), center left (IV), lower left (V) and lower left (VI). It shouldbe noted that low resolution can be used to simulate a much higherresolution, especially when the touch surface is fairly small itself(hand sized). One of the advantages of the system 100 is that it canprovide tactile feedback to more than one distinct point on surface 126at about the same time. Surface 126 can be divided into an array ofactive areas each of which is influenced by a particular actuator. Inthis way, each haptic actuator can directly influence one area, however,effects from other actuators can combine to form a complex signal. Forexample, a finger placed at point X can receive a compound hapticresponse H(X) that can be a function of up to three haptic actuatorsassociated with haptic regions n₁, n₂, and n₃ In other words, thecompound haptic response at point X can be expressed as equation (1):

H(X)=H(n₁, n₂, n₃)   eq (1),

where the relative contributions of each haptic node n₁, n₂, and n₃ isrelated to the distance of X from each haptic node, respectively. Forexample, when at least two fingers are close together get one typesignal can be generated. However when the at least two fingers moveapart, then each of the at least two fingers can feel different hapticsignals from different haptic actuators. Moreover, silent areas orquiescent areas (i.e., regions where a user would feel little or noappreciable sensation) can be created. These silent areas can be createdwhen actuators generate signals that destructively interfere (signalsare 180° out of phase with each other) within the region (also referredto as a null region).

In the case where the haptic actuators are piezo-electric in nature, thevibro-tactile response is generated by the mechanical flexion of amember composed of a piezoelectric material. FIG. 3 schematicallyillustrates the structure and the principle of the operation of arepresentative multiple layer piezoelectric actuator assembly (alsoreferred to as haptic device) 300 in accordance with the invention. Thepiezo-electric assembly 300 comprises an upper piezoelectric layerpiezoelectric 302 and a lower piezoelectric layer 304 (it should benoted that there can be any number of layers). The piezoelectricassembly 300 can either expand or contract in accordance with thedirection of an applied voltage V. For example, by applying, to theupper layer 302, a certain voltage of the direction opposite to thelower layer, the upper layer 302 contracts and the lower layer 304expands at about the same time thereby causing the actuator 300 to bendupward (or downward) as a whole. Due to the inertial coupling of theactuator assembly 300 to the surface 126 when the inertial mass of theactuator is oscillated (by varying the applied voltage V), the inertialsensations are transmitted through member 306 to the surface 126. A foamlayer (or other compliant layer) can be coupled between actuatorassembly 300 and the surface 126 to provide compliance and allow thesurface 126 to move inertially. It should be noted that haptic actuator300 has a fast tactile bandwidth so that it can be used into tactilefeedback device. Haptic actuator 300 can activate with very smalllatency so that it can be used in interactive tactile feedbackapplication and consumes relatively low power and requires very lowvoltage. Advantageously, by using the multi-layered piezoelectric hapticactuator, it is possible to form the haptic device into small and thinchip.

Haptic actuator 300 generates force F directly proportional to voltage Vapplied to the haptic actuator 300 by the controller. Depending upon thepolarity of voltage V, layers 302/304 either expand/contract orcontract/expand resulting in a displacement ΔY of beam member 306. (Itshould be noted that displacement ΔY is much larger than thelongitudinal contraction and expansion of each of layers 304/302.) Morespecifically, the displacement ΔY can be calculated using equation (2):

$\begin{matrix}{{\Delta \; Y} = {k_{1} \times d_{31} \times \left( \frac{L}{t} \right)^{2} \times V}} & {{eq}\mspace{14mu} (2)}\end{matrix}$

-   -   where,    -   k₁: correction factor    -   d₃₁: piezoelectric constant value    -   L: length of the actuator    -   t: thickness of one layer of the actuator    -   V: voltage applied to the actuator

The relationship between force F and voltage V can be quantified byequation (2):

$\begin{matrix}{F = {k_{2} \times d_{31} \times \frac{T}{W \times E \times L}{\,^{}{\times V}}}} & {{eq}\mspace{14mu} (3)}\end{matrix}$

-   -   k₂: correction constant value    -   d₃₁: piezoelectric constant value    -   L: length of the actuator    -   T: thickness of the actuator    -   V: voltage applied to the actuator    -   E: longitudinal elastic coefficient

As illustrated in FIG. 4, haptic device 300 can be used as a pressuresensor simply by sensing a voltage Vp generated by the displacement ΔYof member 306 caused by force F applied to the surface of surface 126.In this way, by monitoring voltage Vp, haptic device 300 can beconfigured to act as an integrated haptic actuator/pressure sensorarranged to change operational modes (passive to active, and viceversa). In the passive mode, the haptic device 300 can act as a pressuresensor by sensing if voltage Vp is generated by haptic device 300 whenforce F impinges on member 306 according to equation (3). When (and if)the sensed voltage Vp exceeds a threshold voltage Vpth, then the hapticdevice 300 can changes modes from passive mode to active mode. In activemode, a haptic profile is used to direct a controller circuit to providevoltage V to member 306 (as shown in FIG. 3) resulting in displacementΔY creating the desired vibro-tactile effect on surface 126 inaccordance with the selected haptic profile. In this way, haptic device300 can considered an integrated pressure sensor/haptic feedback devicethat can be automatically switched back and forth between a pressuresensing device and a haptic feedback device.

Using the arrangements illustrated in FIGS. 3 and 4, pressureinformation can be linked with haptic feedback. For example,vibro-tactile sensation can be increased with increasing pressure, andvice versa. Accordingly, when a user exerts increased pressure (i.e.,presses harder) on a surface, the amount of vibration felt by the userincreases thereby informing the user that they are pressing harder. Inanother embodiment, a touch event can be characterized as either a lighttouch event or a hard touch event. In this situation, differentvibro-tactic sensations can be produced based upon whether the touchevent is light or hard (or something in between). For example, a lighttouch may correspond to a situation where a user uses light touchpressure to slide their finger across a surface whereas a hard press maybe when a user pushes down on the surface with greater pressure, such asa button click. In one example, a hard press initiates a selection andthat results in a click vibration being produced while a light touchperforms no selection but does provide notification to the user that theuser is in a location suitable for making a selection. In anotherembodiment, a click off vibration can be performed when the user beginslifting off of the touch screen i.e., the lift out effect can beproduced as the user is in the process of lifting off of the screeninstead of immediately thereafter. These different touch events can bedistinguished by providing multiple threshold levels or sequence ofthresholds that can be used. For example, a light touch can correspondto a first threshold level whereas a heavy touch can coincide with asecond threshold level, and so on. Moreover, a lift off event can bedistinguished when a sensed voltage indicates that a correspondingapplied pressure is decreasing over time.

FIG. 5 shows a flowchart detailing a process 500 for using apiezoelectric haptic actuator as both a pressure sensor and activehaptic feedback device in accordance with an embodiment of theinvention. Process 500 begins at 502 by sensing a node of hapticactuator 300. If, at 504, a voltage Vp is sensed, then the hapticactuator is in passive mode at 506. In passive mode voltage Vp is beinggenerated by force F applied to the haptic actuator member 306 inaccordance with equation (3) and shown in FIG. 4. At 508, if the sensedvoltage is determined to be greater than a threshold voltage value, thenat 510, the haptic actuator can be considered to be in active mode. Byactive mode, it can be understood to mean that the haptic actuator isnow in a position to receive a voltage from a controlling circuitthereby causing haptic actuator to actively provide haptic feedback inthe form of a vibro-tactile response created by haptic actuator member306 (FIG. 3). Once it is determined that the haptic actuator is inactive mode, then at 512, a haptic profile is retrieved from a hapticprofile database. In the described embodiment, the haptic profile can bebased upon any number of factors. For example, if the haptic actuator isassociated with a button element of a graphical user interface, then thefirst threshold value can be indicative of an amount of pressureconsistent with an intended pressing action by as user as opposed to aglancing touch not intended to be acted upon as a button press. At 514,once the appropriate haptic profile has been retrieved, a controllingcircuit applies an appropriate control voltage to the haptic actuatorconsistent with the haptic profile. In some cases, the haptic profilecan vary depending upon the amount of pressure applied. If, at 516, ithas been determined that the haptic response is complete, then thehaptic actuator is set to passive mode at 518 and control is passed backto 502.

Additional embodiments of the present invention are presented inaccordance with FIGS. 6-13.

FIG. 6 shows display device 112 displaying representative haptic activeGUI elements in accordance with an embodiment of the invention. Itshould be noted that although only a small number of all possible GUIelements are discussed herewith, the following discussion can apply toany appropriate haptic active GUI element. Accordingly, in the contextof device 100, processor 104 can direct display device 112 to display anumber of haptic active GUI elements that taken together formrepresentative GUI 600. Such haptic active GUI elements can include, forexample, first type haptic active button icon 602, haptic active slidericon 604 having movable slider element 606, keypad 608 formed of anumber of second type haptic active buttons 610. In the context of thisdiscussion, the designations first and second type indicate that thehaptic response associated with a particular GUI element. For example, afirst type haptic response can be a high frequency vibration whereas asecond type haptic response can palpable click. In should also be notedthat button elements can be any appropriate shape or size.

FIG. 7 shows rectangular button element 702 and circular button element704. In the described embodiments, button elements 702 and 704 can bearranged to exhibit haptic response H_(button)(x) that enables a user toaccurately identify target areas 706 and/or 708, respectively (alsoreferred to as “sweet” spots). Target areas 706 and/or 708 can representthose portions of button elements 702 and 704 that are most sensitive tothe application of pressure by a user and therefore would provide anoptimum haptic feedback response. For example, H_(button)(x) can beconfigured to provide haptic feedback that depends upon the location ofa user's finger within a region defined by the button element.Accordingly, the button element can be divided into separate regionseach having its own haptic response or a haptic response related to thehaptic response of the other regions. In this way, the varying hapticresponses can be arranged to “lead” a user to a particular target area706 or 708. For example, button 702 (as well as button 704) can bedivided into regions “target”, region 1, and region 2. A user canexperience a different haptic response in each of the regions such asH_(target) in target region (a fast vibration, for example), H₁ inregion 1 (a slower vibration than that presented in the target region),H₂ in region 2 and so on. The various haptic responses can be arrangedto provide the user with the ability to “feel” their way to the targetarea without the need to actually view the display 112. In this way, thedescribed embodiments allows a user to pinpoint the location of a targetarea of a button element (or any other GUI element so configured) indark or low light conditions or in situations where the user cannoteasily view display 112 by “feeling” their way to the particular targetarea. Accordingly, by following the “lead” provided by the varyinghaptic responses, a user can accurately identify target areas ofparticular buttons (or GUI elements) and once identified, can provideaccurate tactile inputs to device 100 with the need to actually viewdisplay device 112. For example, by appropriately configuring hapticactive button elements 610, a user can accurately enter data (such as aphone number) into device 100 by way of keypad 608 using thevibro-tactile feedback provided by haptic active button elements 610.

As shown in FIG. 8A, button elements 702 or 704 can have an associatedhaptic profile H(x). In one implementation, the haptic profile H(x) caninclude profiles that are related to each other in such a way as to leada user to a particular location within the button (such as the targetarea). For example, as the user's finger moves across the portion ofdisplay 112 having keypad 608, the user can experience varied hapticresponses that can depends upon the particular location of the user'sfinger on the surface 126. In this way, the user can be “lead” to thetarget area 706 by feeling his way along the surface 126. As shown inFIG. 8B, as the user's finger is moved across keypad 608, the userexperiences haptic feedback based upon haptic profile H(x). In theexample shown, in the region between button elements 602, the user'sfinger would be moving across a portion of surface 126 associated with aquiescent haptic response. By quiescent it is meant that the user wouldfeel little or no haptic effects. However, once the user's fingerintersects outer boundary 802 (that could be visible, or not) of buttonelement 702, the user experiences haptic sensation H₁ corresponding toregion 1. As the user's finger moves from region 1 to region 2, the userexperiences haptic sensation H₂ corresponding to region 2, and so onuntil the user reaches target area 708. If the user's finger continuesto move in such a way that it would be leaving target area 708 byre-entering region 2, for example, then the user would be made aware ofthis fact since the user would experience a change in the hapticsensation indicating that the user's finger is leaving target area 708.

FIG. 9A shows slider element 900 as a particular embodiment of sliderelement 604 shown in FIG. 6. Slider element 900 includes slider portion902 that can be moved across surface 126 within the confines of sliderelement 900 by the action of a user's finger. As the user's finger isplaced in contact with slide portion 902, the user can experience ahaptic sensation consistent with haptic profile H_(slider)(x) shown inFIG. 9B. In the described embodiment, as the user's finger (while stillin contact with slide portion 902) moves slide portion 902 from left toright (or vice versa), then the user can experience vibro-tactilefeedback based upon haptic profile H_(slider)(x) where, for example, thehaptic sensations monotonically increase from left to right andmonotonically decrease from right to left, or vice versa. In some cases,H_(slider)(x) can be a linear function of touch co-ordinate x (as shownin FIG. 9B) or can be non-linear depending of course on the applicationfor which slider 900 is being used.

FIGS. 10A-10B illustrates yet another embodiment whereby a user can“feel” an edge of image feature 612 presented on display 112 shown inFIG. 6. In this embodiment, the user can “feel” edge E of feature 612due to the fact that the haptic profile H_(edge)(x) provides a hapticresponse in a region R in proximity to the edge E of feature 612. Insome cases, the haptic response H_(edge)(x) can be asymmetric (as shownin FIG. 10B) where a user approaching edge E from an interior portion offeature 612 will feel a gradually intensifying haptic responseindicating to the user that edge E is being approached from the interiorof feature 612. On the other hand, if the user's finger is approachingedge E of feature 612 from a region exterior to feature 612, then thehaptic response H_(edge)(x) will cause the user to experience a sharpstep increase in haptic sensation at or near edge E itself. In the casewhere the haptic profile H_(edge)(x) is symmetric, then the user willexperience a haptic sensation of substantially equal no matter if edge Eis approached from the interior or exterior regions of feature 612.

FIG. 11 is a diagram of a zoom gesture method 1100 in accordance with anembodiment of the invention. The zoom gesture can be performed onmulti-touch multi-haptic surface 126. It should be noted that initially,the haptic devices associated with the surface 126 can be in the passivestate whereby the controller can monitor the condition of each of thehaptic nodes by determining if there is voltage Vp is being generated bythe haptic device indicative of pressure being applied in the vicinityof the haptic node. Accordingly, the zoom gesture method 1100 generallybegins at block 1102 where the presence of at least a first finger and asecond finger are detected on a touch sensitive surface of the surface126 at about the same time. In the described embodiment, the nature ofthe multi-touch event can be determined based upon either the presenceof at least two fingers indicating that the touch is gestural (i.e.,multi-touch) rather than a tracking touch based on one finger and/or bythe pressure asserted by the fingers on the surface 126. The pressureasserted by the fingers on the touch screen can be determined bymonitoring the voltage Vp described above. If it is determined at block1104 that the presence of the two fingers represents a gesture, then thehaptic devices nearest the touch point are set to active mode in orderto provide a vibro-tactile response at 1106 to each of the fingersduring the gesture. In the described embodiment, the vibro-tactileresponse provided to each finger can have the same profile or differentprofiles. For example, if it the pressure applied by one finger issubstantially greater than that applied by the other finger, then thevibro-tactile response for the two fingers can be different due to thevarying pressure applied by each finger. For the most part, however, thehaptic profiles will be correlated to assure that the user has aperception that the haptic effects are consonant with each other.Furthermore, it should be noted that in some cases, the presence of onlytwo fingers indicates that the touch is a gestural touch. In othercases, any number of more than two fingers indicates that the touch is agestural touch. In fact, the gestural touch may be configured to operatewhether two, three, four or more fingers are touching, and even if thenumbers change during the gesture, i.e., only need a minimum of twofingers at any time during the gesture.

Following block 1106, the zoom gesture method 1100 proceeds to block1108 where the distance between at least the two fingers is compared.The distance may be from finger to finger or from each finger to someother reference point as for example the centroid. If the distancebetween the two fingers increases (spread apart) at 1110, a zoom-insignal is generated at 1112, otherwise a zoom out signal is generated atblock 1114. The zoom-in signal, in turn, causes the haptic devicesassociated with the two fingers to provide a zoom-in haptic signal at1116. Such a zoom in haptic signal can be, for example, faster (orslower) or more intense (or less intense) vibration as the distancebetween the two fingers increases. If the distance between two fingersdecreases (close together), the zoom-out signal generated at block 1114can cause the haptic devices associated with the two fingers to providea zoom out haptic signal at 1118.

In most cases, the set down of the fingers will associate or lock thefingers to a particular GUI object being displayed. For example, thetouch sensitive surface can be a touch screen, and the GUI object can bedisplayed on the touch screen. This typically occurs when at least oneof the fingers is positioned over the GUI object. As a result, when thefingers are moved apart, the zoom-in signal can be used to increase thesize of the embedded features in the GUI object and when the fingers arepinched together, the zoom-out signal can be used to decrease the sizeof embedded features in the object. The zooming typically occurs withina predefined boundary such as the periphery of the display, theperiphery of a window, the edge of the GUI object, and/or the like. Insome cases, a haptic effect can be provided giving the user a warningthat the predefined boundary is being approached. The embedded featuresmay be formed on a plurality of layers, each of which represents adifferent level of zoom. In most cases, the amount of zooming and theassociated haptic effect varies according to the distance between thetwo objects. Furthermore, the zooming typically can occur substantiallysimultaneously with the motion of the objects. For instance, as thefingers spread apart or closes together, the object zooms in or zoomsout at the same time and the corresponding haptic effect will change.Although this methodology is directed at zooming, it should be notedthat it may also be used for enlarging or reducing. The zoom gesturemethod 1100 may be particularly useful in graphical programs such aspublishing, photo, and drawing programs.

FIGS. 12A-12H illustrate a zooming sequence using the method describedabove. FIG. 12A illustrates a display presenting a GUI object 1202 inthe form of a map of North America with embedded levels which can bezoomed. In some cases, as shown, the GUI object is positioned inside awindow 1204 that forms a boundary of the GUI object 1202. Also shown arethe haptic profiles for each of the fingers relating the distance dbetween the two fingers to the corresponding haptic response H(d)experienced at each finger. It should be noted that in this example, themagnitude of the haptic response H(d) at each finger is denoted by thesize of the circle for each response. In this case, as the distancebetween the two fingers increases, the haptic effect H for each fingerincreases linearly with distance d. For example, when the two fingersare close together as in FIG. 12B, the haptic effect H is quite small asevidenced by the small size of the circle whereas as the two fingersmove apart, the haptic effect H becomes progressively stronger at eachfinger. It should be noted that for the sake of simplicity only, thehaptic profile H(d) is presumed linear for zooming in/out and non-linearfor the rotation gesture shown in FIG. 12F. In the described embodiment,as the zoom factor increases, the haptic profile H(d) can change by, forexample, the slope becoming more steep as the resolution of theunderlying map increases as shown in FIG. 12G. FIG. 12B illustrates auser positioning their fingers 1206 over a region of North America 1202,particularly the United States 1208 and more particularly California1210. In order to zoom in on California 1210, the user starts to spreadtheir fingers 1206 apart as shown in FIG. 12C. As the fingers 1206spread apart further (distance increases) the haptic effect felt by thetwo fingers changes as the map zooms in further on Northern California1212, then to a particular region of Northern California 1214, then tothe Bay area 1216, then to the peninsula 1218 (e.g., the area betweenSan Francisco and San Jose Area), and then to the city of San Carlos1220 located between San Francisco and San Jose as illustrated in FIGS.12D-12H. In order to zoom out of San Carlos 380 and back to NorthAmerica 368, the fingers 366 are closed back together following thesequence described above, but in reverse (along with the correspondinghaptic effect).

FIG. 13 is a diagram of a GUI operational method 1300, in accordancewith one embodiment of the present invention. The method generallybegins at block 1302 where a virtual scroll wheel is presented on thedisplay. In some cases, the virtual scroll wheel can include a virtualbutton at its center. The virtual scroll wheel is configured toimplement scrolling as for example through a list and the button isconfigured to implement selections as for example items stored in thelist. Following block 1302, the method proceeds to block 1304 where thepresence of at least a first finger and more particularly, first andsecond fingers (to distinguish between tracking and gesturing) over thevirtual scroll wheel is detected on a touch screen. The touch screen ispositioned over or in front of the display. By way of example, thedisplay can be an LCD and the touch screen can be a multi-touch touchscreen. Following block 1304, the method proceeds to block 1306 wherethe initial position of the fingers on the virtual scroll wheel is set.By way of example, the angle of the fingers relative to a referencepoint can be determined (e.g., 12 o clock, 6 o clock, etc.).

Following block 1306, the method 1300 proceeds to block 1308 where arotate signal is generated when the angle of the fingers change relativeto the reference point. In most cases, the set down of the fingersassociate, link or lock the fingers (or finger) to the virtual scrollwheel when the fingers are positioned over the virtual scroll wheel. Asa result, when the fingers are rotated, the rotate signal can be used torotate the virtual scroll wheel in the direction of finger rotation(e.g., clockwise, counterclockwise) at 1310 as well as provide anaudible as well as palpable “click” at 1312 using at least two hapticactuators at 1310 to provide a physical sensation at the two fingersconcurrently with the audible click simulating the “feel” of the click.In most cases, the amount of wheel rotation varies according to theamount of finger rotation, i.e., if the fingers move 5 degrees then sowill the wheel. Furthermore, the rotation typically occurs substantiallysimultaneously with the motion of the fingers. For instance, as thefingers rotate, the scroll wheel rotates with the fingers at the sametime.

The various aspects, embodiments, implementations or features of theinvention can be used separately or in any combination. The invention ispreferably implemented by hardware, software or a combination ofhardware and software. The software can also be embodied as computerreadable code on a computer readable medium. The computer readablemedium is any data storage device that can store data which canthereafter be read by a computer system. Examples of the computerreadable medium include read-only memory, random-access memory, CD-ROMs,DVDs, magnetic tape, optical data storage devices, and carrier waves.The computer readable medium can also be distributed overnetwork-coupled computer systems so that the computer readable code isstored and executed in a distributed fashion.

While this invention has been described in terms of several preferredembodiments, there are alterations, permutations, and equivalents, whichfall within the scope of this invention. For example, although theinvention has been primarily directed at touchscreens, it should benoted that in some cases touch pads can also be used in place oftouchscreens. Other types of touch sensing devices can also be utilized.It should also be noted that there are many alternative ways ofimplementing the methods and apparatuses of the present invention. It istherefore intended that the following appended claims be interpreted asincluding all such alterations, permutations, and equivalents as fallwithin the true spirit and scope of the present invention.

1. An apparatus providing multi-touch haptic feedback, comprising atouch pad having a touch sensitive surface arranged to receive a userprovided multi-touch event associated with at least two differentlocations on the touch sensitive surface; a multi-touch detectionmechanism operatively coupled to the touch sensitive surface thatdetects the multi-touch event and generates a corresponding amulti-touch signal; and a plurality of haptic feedback devicesoperatively coupled to the multi-touch detection mechanism and the touchsensitive surface cooperatively arranged to concurrently provide tactilefeedback at each of the at least two different locations on the touchsensitive surface in response to the multi-touch signal.
 2. Theapparatus as recited in claim 1, wherein the tactile feedback at each ofthe at least two different locations are discreet from one another. 3.The apparatus as recited in claim 2, wherein when the multi-touch signalindicates that the multi-touch event is a dynamic multi-touch eventindicating a change in the multi-touch event, then the tactile feedbackat each of the at least two different locations is updated to reflectthe change in the multi-touch event.
 4. The apparatus as recited inclaim 1, wherein the tactile feedback event is different for each of theat least two different locations.
 5. A method for processing a touchevent at a touch sensitive surface, comprising: receiving a userprovided multi-touch event associated with at least two differentlocations on the touch sensitive surface; detecting the multi-touchevent; generating a multi-touch signal corresponding to the multi-touchevent; and concurrently providing tactile feedback at each of the atleast two different locations on the touch sensitive surface in responseto the multi-touch signal wherein the tactile feedback at each of the atleast two different locations are discreet from one another.
 6. Themethod as recited in claim 5, wherein when the multi-touch signalindicates that the multi-touch event is a dynamic multi-touch eventcorresponding to a change in the multi-touch event over a period oftime, then the tactile feedback at each of the at least two differentlocations is updated in real time to reflect the change in themulti-touch event.
 7. A multi-touch haptic mechanism, comprising: atouch pad having a touch sensitive surface arranged to detect a usertouch event at substantially any location on the touch sensitivesurface; and a plurality of independent haptic devices operativelycoupled to the touch sensitive surface each providing a correspondingtype of tactile feedback in response to a nearby user touch eventthereby providing a tactile feedback at substantially any location onthe touch sensitive surface at which the user touch event has occurred.8. The multi-touch haptic mechanism as recited in claim 7, wherein eachof the nearby haptic device only responds to the user touch event in oneor more associated regions of the touch sensitive surface.
 9. Themulti-touch haptic mechanism as recited in claim 8, wherein at least twonearby haptic devices concurrently respond to the user touch event byproviding a compound haptic response.
 10. The multi-touch hapticmechanism as recited in claim 9 wherein the compound haptic response isdifferent than the haptic response type provided by either of the atleast two independent haptic devices separately.
 11. A method forprocessing a user touch event at a touch sensitive surface having aplurality of independent haptic devices coupled thereto, wherein each ofthe haptic devices provides a corresponding type of tactile feedback,comprising: detecting the user touch event on the touch sensitivesurface; and providing a particular tactile feedback in response to theuser touch event by the haptic device associated with the location ofthe user touch event, wherein the particular tactile feedback isdifferent for each location on the touch sensitive surface.
 12. Themethod as recited in claim 11, wherein each of the plurality ofindependent haptic devices only responds to the user touch event in oneor more associated regions of the touch sensitive surface.
 13. Anintegrated device arranged to act as both a force sensing device and ahaptic feedback device, comprising: a touch sensitive surface; acontroller unit; and a mechanical actuator coupled with the controllerunit and the touch sensitive surface, wherein the integrated device actsas the force sensing device by, generating an output voltage in directproportion to a force applied to the mechanical actuator by a usertouching the touch sensitive surface, sensing the output voltage by thecontroller unit, and converting the sensed output voltage to anindication of the applied force, and wherein only when the sensed outputvoltage exceeds a voltage threshold level does the integrated device actas the haptic feedback device by, halting the sensing of the outputvoltage by the controller unit, activating the mechanical actuator bythe controller unit, and imparting a physical force to the touchsensitive surface by the activated mechanical actuator.
 14. The deviceas recited in claim 13, wherein the imparted physical force creates avibro-tactile response felt by the user commensurate with the physicalforce applied by the user on the touch sensitive surface.
 15. Anintegrated method for both sensing a force applied to a touch sensitivesurface and providing a haptic feedback response to the sensed force byan integrated device having a controller unit coupled to the touchsensitive surface and a mechanical actuator, comprising: wherein theintegrated device senses the applied force by, generating an outputvoltage in direct proportion to the force applied to the mechanicalactuator, sensing the output voltage by the controller unit, andconverting the sensed output voltage to an indication of the appliedforce; and wherein the integrated device provides the haptic feedbackresponse by, if the sensed output voltage exceeds a voltage thresholdlevel, then halting the sensing of the output voltage by the controllerunit, activating the mechanical actuator by the controller unit, andimparting a physical force to the touch sensitive surface by theactivated mechanical actuator.
 16. The method as recited in claim 15,wherein the imparted physical force creates a vibro-tactile responsefelt by the user commensurate with the physical force applied by theuser on the touch sensitive surface.
 17. The method as recited in claim15, wherein the mechanical actuator operatively coupled to the touchsensitive surface is arranged to provide a specific type of tactilefeedback corresponding to the amount of pressure applied to the touchsensitive surface by the user.
 18. An electronic device, comprising: atouch pad having a touch sensitive surface arranged to process a usertouch event; and a plurality of haptic feedback devices each of which isoperatively coupled to the touch sensitive surface and each respondingto the user touch event only in a specific region of the touch sensitivesurface and arranged to provide tactile feedback singly or incombination with others of the plurality of haptic feedback devices inresponse to the user touch event.
 19. The electronic device as recitedin claim 18, wherein when the touch sensitive regions of at least two ofthe plurality of haptic devices overlap and if the user touch eventoccurs in the overlapping region, then the at least two haptic devicescooperate to provide a combined haptic feedback response.
 20. Theelectronic device as recited in claim 19, wherein the combined hapticfeedback response includes a first component associated with one of theat least two haptic devices and a second component associated with theother of the at least two haptic devices.
 21. The electronic device asrecited in claim 20, wherein the first component is related to adistance from the user touch event to the one of the at least two hapticdevices.
 22. The electronic device as recited in claim 20, wherein thesecond component is related to a distance from the user touch event tothe other of the at least two haptic devices.
 23. The electronic deviceas recited in claim 20, wherein the first and the second componentscombine linearly to form the compound haptic response.
 24. An electronicdevice, comprising: a touch pad having a touch sensitive surfacearranged to receive a user touch event provided by a user; a controllercoupled and in communication with the touch pad arranged to at leastanalyze the user touch event and/or a state of the touch pad and basedupon the analysis provide a user touch event signal in response to theuser touch event; and at least one haptic device operatively coupled tothe controller arranged to receive the user touch event signal, whereinthe at least one haptic device responds to the user touch event signalby providing an appropriate haptic feedback response to the user basedupon the analysis provided by the controller.
 25. The electronic deviceas recited in claim 24, wherein the touch sensitive surface is arrangedto receive different types of user touch events each being characterizedby an amount of pressure applied on the touch sensitive surface by auser.
 26. The electronic device as recited in claim 25, wherein thehaptic device responds to the user touch event based upon the appliedpressure.